Millions of Americans receive psychosocial interventions such as psychotherapy each year for treatment of mental health, behavioral health, and addiction problems. Evidence-based psychotherapies such as cognitive-behavioral therapy (CBT) are increasingly emphasized in professional practice guidelines, taught in training programs, sought out by consumers, and valued by payers. Yet, at present, there is no scalable method for evaluating the quality of psychotherapy services. In research settings, human-based behavioral coding methods are used, but these are time consuming, costly, and rarely used in real-world clinical settings. Psychotherapy is a ?talk based? treatment, and as such, the active ingredients and quality of the therapy are found in the spoken language and interaction of clinician and client. The current proposal will develop a software system (CORE-CBT) that will automatically generate a summary report of quality metrics for CBT, from an audio recording of a CBT session, using speech signal processing and machine learning. Importantly, the current work builds from previous, successful work in developing CORE-MI, an automated system for evaluating motivational interviewing for addiction. The existing CORE-MI tool and the expertise gained through its development will serve as the foundation for the current proposal. Our technical team is complemented by clinical and implementation expertise via the partnership of the Beck Community Initiative (BCI) with Philadelphia?s DBHIDS. Since 2007, BCI has partnered in training staff at 60 programs in the DBHIDS network to deliver high-quality CBT, including objective quality ratings via the Cognitive Therapy Rating Scale (CTRS). Through ongoing training efforts, the BCI has generated an archive of over 6,500 recorded CBT sessions, of which more than 2,300 have been CTRS-coded for quality. This session archive and ongoing real-world trainings in the DBHIDS network provide the perfect context to develop CORE-CBT, a tool that automates quality coding for CBT. Aim 1: ?Extend and evaluate machine learning algorithms to automatically code CBT competence, using an archive of more than 2,000 CBT fidelity-coded sessions. ?Aim 2: ?Adapt and enhance front-end user interface and session reports with key stakeholders (frontline clinicians, supervisors, and clinic administrators), evaluating acceptability, appropriateness, and feasibility. ?Aim 3: ?Evaluate CORE-CBT technology during CBT training in 4 BCI partner clinics in the DBHIDS network, with approximately 48-64 frontline clinicians and 4,000 sessions, assessing both implementation outcomes (adoption, sustainment) and service outcomes (increased capacity, efficiency, and effectiveness of training). ?Aim 4:? Evaluate the appropriateness of CORE-CBT and its metrics for quality assurance (clinics) and value-based reimbursement (payers), using an in-person participatory design workshop. An advisory board of behavioral health payers will provide feedback over the course of the study to maximize generalizability and future uptake in large healthcare systems.